1. Field of the Invention
The present invention relates to a technique of correcting blurring of a captured image caused by camera shake by using motion vector detection information obtained from blurring of a moving image.
2. Description of the Related Art
An image capturing apparatus such as a video camera which captures a moving image suffers image blurring owing to camera shake when the lens zooms to the telephoto end. To prevent image blurring caused by camera shake, there has conventionally been proposed a technique of detecting the motion vector of an image from a captured image signal and correcting image blurring based on the motion vector.
Known examples of the conventional method of detecting the motion vector of a moving image are a correlation method based on correlation calculation, and block matching.
According to block matching, an input image signal is divided into a plurality of blocks (e.g., eight pixels×eight lines) of a proper size. Differences from pixels in a predetermined range of a preceding field (or frame) are calculated for each block. A block of a preceding field (or frame) in which the sum of the absolute values of the differences becomes minimum is searched for. The relative shift between frames represents the motion vector of the block.
Matching calculation is discussed in detail in Morio Onoe, et al., “Information Processing”, Vol. 17, No. 7, pp. 634-640, July 1976.
An example of a conventional motion vector detection method using block matching will be explained with reference to FIG. 7. FIG. 7 is a schematic block diagram of an apparatus which prevents blurring according to a conventional motion vector detection method.
An image signal (field or frame) subjected to motion vector detection is input via an input terminal 101 to an image memory 110 and a filter 102 for extracting the spatial frequency. The image memory 110 temporarily stores the image signal. The filter 102 extracts, from the image signal, a spatial frequency component useful for motion vector detection. That is, the filter 102 removes the low and high spatial frequency components of the image signal.
The image signal having passed through the filter 102 is input to a binarization circuit 103. The binarization circuit 103 binarizes the image signal using zero level as a reference. More specifically, the binarization circuit 103 outputs the sign bit of the output signal.
The binary image signal is input to a correlation calculation circuit 104 and a memory 105 serving as a 1-field period delay means. The correlation calculation circuit 104 further receives an image signal of a preceding field from the memory 105.
According to block matching, the correlation calculation circuit 104 divides the image area into a plurality of block areas of a proper size, as described above. The correlation calculation circuit 104 calculates the correlation between the current and preceding fields for each block, and outputs the resultant correlation value to a motion vector detection circuit 106. The motion vector detection circuit 106 detects the motion vector of each block from the calculated correlation value. More specifically, the motion vector detection circuit 106 searches for a block of a preceding field having a minimum correlation value. The motion vector detection circuit 106 detects the relative shift between the blocks of the current and preceding fields as a motion vector quantity.
The motion vector quantity of each block is input to a motion vector determination circuit 107. The motion vector determination circuit 107 determines the motion vector quantity (representative vector quantity) of an entire image from the motion vectors of respective blocks. More specifically, the motion vector determination circuit 107 determines the median or average of motion vector quantities of respective blocks as the motion vector quantity of the entire image. An integrating circuit 108 integrates the motion vector quantity of the entire image obtained by the motion vector determination circuit 107, converting it into an integrated motion vector quantity.
A memory read control circuit 109 controls the readout position in the image memory 110 so as to cancel image blurring in accordance with the integrated motion vector quantity. Then, the image memory 110 outputs a blurring-corrected image signal.
A representative vector detection circuit 21 includes the filter 102, binarization circuit 103, correlation calculation circuit 104, memory 105, motion vector detection circuit 106, and motion vector determination circuit 107.
FIG. 8 is a block diagram showing a state in which the above-described arrangement is assembled into an image capturing apparatus such as a video camera.
In FIG. 8, the image capturing apparatus comprises a motor-driven zoom lens made up of a zoom lens 10, a zoom switch 33 operated by the user to drive the zoom lens 10 and change the zoom ratio, a zoom control circuit 32 which controls a zoom motor in accordance with the state of the zoom switch 33, a zoom motor 31, and a zoom encoder 34 which detects the position of the zoom lens 10. The position of the zoom lens 10 changes in accordance with the operation of the zoom switch 33. More specifically, it is controlled to move the zoom lens 10 to a target position by driving the zoom motor 31 via the zoom control circuit 32 in accordance with zoom position information input from the zoom switch 33 so that the zoom encoder 34 obtains a value matching the position information.
An object image formed via the zoom lens 10 and a main optical system 11 on the light receiving surface of an image sensor 12 formed from a CCD or the like is converted into an electrical signal. A camera signal processing circuit 13 converts the electrical signal into a standard video signal or the like as an image signal.
The video signal obtained by the camera signal processing circuit 13 is input to the image memory 110, and also to the representative vector detection circuit 21. The representative vector detection circuit 21 outputs the motion vector quantity of an entire image. The integrating circuit 108 integrates the motion vector quantity of the entire image obtained by the representative vector detection circuit 21, converting it into an integrated motion vector quantity. The memory read control circuit 109 controls the readout position in the image memory 110 so as to cancel image blurring in accordance with the integrated motion vector quantity. A blurring-corrected image signal is output from the image memory 110, and recorded by a recorder 17.
A reference related to this technique is Japanese Patent Laid-Open No. 7-177425.
When an image capturing apparatus having the above-described arrangement executes a zoom operation, the following problems arise.
(1) When zoom control is performed to change the focal length from a large value to a small one, the correction angle corresponding to the integrated motion vector quantity increases upon the change of the focal length.
(2) To the contrary, when zoom control is performed to change the focal length from a small value to a large one, the correction angle corresponding to the integrated motion vector quantity decreases upon the change of the focal length.
This phenomenon will be explained with reference to FIG. 9.
FIG. 9 is a graph showing an example of the angular displacement and the image shift from the center of the optical axis on the image capturing plane with respect to the focal length. Even if the angular displacement is constant, the image shift from the center of the optical axis on the image capturing plane changes in accordance with the focal length. A characteristic 47 in FIG. 9 represents an image shift from the center of the optical axis in accordance with the focal length for an angular displacement of 1 deg from the optical axis. Similarly, characteristics 46 and 45 represent image shifts from the center of the optical axis in accordance with the focal length for angular displacements of 2 deg and 3 deg.
A case where the zoom lens is driven to change the focal length of the lens from a (long focal length) to b (short focal length) will be examined.
Even if no vector quantity is detected (the vector quantity is 0) upon a change of the angle of view in zoom driving, when the motion vector quantity integrated by the integrating circuit 108 indicates a shift 41 on the image capturing plane at a focal length a, the shift 41 becomes equal to a shift 42 at a focal length b as a result of the zoom operation. When the shift is converted into a change of the angle from the optical axis, the 1-deg angle represented by the characteristic 47 is replaced with the 3-deg angle represented by the characteristic 45.
The relationship between the image shift from the center of the optical axis on the image capturing plane, the angle (angular displacement) from the optical axis, and the focal length is generally given byl=f×tan θ
l: image shift from the center of the optical axis
f: focal length
θ: angular displacement from the optical axis
That is, the zoom operation changes the camera shake correction angle. The zoom operation impairs the continuity of the correction angle, so a change between images during and after zooming becomes unnatural.
The unnatural change becomes more conspicuous as the focal length changes much more in the zoom operation and as the camera shake correction angle is larger in the zoom operation.
Further, a vector quantity detection error occurs when the object moves upon a change of the angle of view in the zoom operation without any camera shake.
A case where an image shown in FIG. 10A is captured by increasing the focal length (from the wide-angle side to the telephoto side) by the zoom operation will be exemplified.
In FIG. 10A, a captured image 201 within the image capturing range is read out from the image sensor. A block 202 is obtained by dividing the obtained captured image into blocks in order to perform block matching, as described above. Reference numerals 211 and 212 denote objects.
The captured image changes into one shown in FIG. 10B as a result of the zoom operation to set an area 204 shown in FIG. 10A to the image capturing angle of view.
Even if the captured image is free from blurring caused by camera shake or the like in the zoom operation, vector quantities as shown in FIG. 11 are detected. In FIG. 11, arrows 222 indicate detected vector quantities along with movement of the object 211 and that of the object 212 upon a change of the angle of view. Points 223 represent blocks in which no vector quantity is detected because no object exists (matching fails).
In FIG. 11, the motion vector quantity of an entire image obtained by the motion vector determination circuit 107 is a vector quantity indicated by an arrow 221 when calculating the median or average of motion vectors of respective blocks.
This means a motion vector quantity detection error caused by movement of an object upon a change of the angle of view in the zoom operation without any camera shake.
The detected vector quantity changes depending on the zoom speed. As a change of the angle of view per unit time is larger, a larger vector quantity is detected. That is, when the zoom speed is high, the vector detection error becomes large, and the malfunction is more frequently caused by the detection error.